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ABSTRACT 



This report contains the second year evaluation of the 
Student Assistance Program (SAP) in Pennsylvania. These school -based and 
school -linked programs address barriers to learning for youth at risk for 
social and emotional problems, drug and alcohol use and abuse, and 
depression. Second year evaluation focused on identifying essential 
components related to SAP, delineating indicators for each component area, 
and outlining a benchmarking process for technology transfer and ongoing 
improvement. The six areas include: (1) referral processes; (2) parent 

consent practices; (3) intervention; (4) contract provider assessments; (5) 
school-based probation officers' SAP involvement; and (6) SAP improvement and 
satisfaction. The evaluation produced three outcomes. First, nine SAP 
component areas and corresponding indicators were identified. Second, the 
evaluation findings suggest a strategy to improve SAP effectiveness and best 
practices at the school level. Third, a handbook was produced to summarize 
and present the salient findings of the evaluation in a user-friendly format. 
The handbook provides all SAP participants with a means to monitor and 
improve their programs. This report includes the following sections: (1) 

Introduction; (2) Methodology; (3) Results; (4) Incorporation of Components 
and Indictors Results to the Field; and (5) Technology Transfer. (Contains 2 
appendixes; 6 tables; 1 figure; and 8 resources.) ( JDM) 
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Student Assistance Program in Pennsylvania Evaluation 

Executive Summary 



In July of 1999, the Pennsylvania Commission on Crime and Delinquency and the Department of 
Education funded the second year of the evaluation of Student Assistance Programs (SAP) in 
Pennsylvania. Year two efforts focused on identifying essential components related to SAP, 
delineating indicators for each component area, and outlining a benchmarking process for 
technology transfer and ongoing improvement of SAP. The evaluation produced three significant 
outcomes. 

First, using a modified Delphi process, nine SAP component areas and corresponding indicators 
were identified. The process included a literature review, a survey and focus group process with 
the statewide student assistance advisory committee, SAP Commonwealth approved trainers, 
Pennsylvania Association of Student Assistance Professionals conference participants, members 
of county coordination teams and SAP district councils, and school site visits. 

Second, the evaluation findings suggest a strategy to improve SAP effectiveness. The findings 
indicate that increased training and development of SAP stakeholders in Components Four 
(Parent Participation) and Seven (Follow-up and Support) will have a positive impact on SAP 
performance and best practices at the school level. 

Third, a Components and Indicators Handbook was produced to summarize and present the 
salient findings of the evaluation in a user-friendly format. The Handbook provides all SAP 
stakeholders with a concise, informative, and usable means to monitor and improve their 
programs. The centerpiece of the Handbook is the Components and Indicators Checklist that can 
serve as an instrument for assessing, developing, and benchmarking SAP programs. Furthermore, 
we present strategies for using the components and indicators to formulate an action improvement 
plan. The Components and Indicators Handbook is available on the Department of Education 
website: http://www.sap.state.pa.us/ 



STUDENT ASSISTANCE PROGRGAM IN PENNSYLVANIA 
EVALUATION FINAL REPORT 



Submitted to: 

Pennsylvania Commission on Crime and Delinquency 
P.O.Box 1167 

Harrisburg, Pennsylvania 17108-1167 
Telephone: 800-692-7292 
Fax:717-783-7713 

October, 2000 

Prepared by 

Carl I. Fertman, Ph.D. and Jo Schlesinger, M.Ed. 

University of Pittsburgh 
5D21 Wesley W. Posvar Hall 
Pittsburgh, Pennsylvania 15260 
Telephone: 412-648-7191 
Fax:412-648-7198 
carl@pitt.edu / jschlesi@pitt.edu 

Cele Fichter, M.P.M., and Susan Tarasevich, M.Ed. 

St. Francis Institute for Psychiatric and Addiction Services 
40045 th Street 

Pittsburgh, Pennsylvania 15201 
Telephone: 412-622-4526 
Fax: 412-622-6756 

fichterc@sfhs.edu / tarasevichs@sfhs.edu 

Xiaoyan Zhang, Ph.D. and Holly Wald, Ph.D. 

KIT Solutions, Inc. 

105 Braunlich Drive 
Pittsburgh, Pennsylvania 15237 
Telephone: 412-366-7188 
Fax:412-366-7199 

xiaoyanz@kitsolutions.net / hollyw@kitsolutions.net 







5 



TABLE OF CONTENTS 



Student Assistance Program in Pennsylvania 
Evaluation Final Report 

SECTION ONE: INTRODUCTION 1 

Background 1 

Summary of Year One Results 1 

Aims of Year Two 2 

SECTION TWO: METHODOLOGY FOR DEVELOPING COMPONENTS 

AND INDICATORS 3 

Literature Review 3 

Statewide Student Assistance Advisory Committee 14 

Survey of Commonwealth Approved SAP Trainers 14 

Stakeholder Focus Groups 14 

School Site Visits:Exemplary and Struggling 15 

SECTION THREE: RESULTS 17 

Survey of Commonwealth Approved SAP Trainers 17 

Focus Group Results 22 

Site Visit Results: Exemplary Schools 24 

Site Visit Results: Struggling Schools 25 

SECTION FOUR: INCORPORATION OF COMPONENTS AND INDICATORS 

RESULTS TO THE FIELD 27 

Mapping the Data 27 

Description of Databases From Year One Evaluation 30 

Procedure For Linking Two Databases 31 

Findings 33 

SECTION FIVE : TECHN OLOG Y TRANSFER 36 

Components and Indicators Handbook 36 



1 

I 

I 

I 

f 

I 

I 

1 

I 

1 

I 

I 

§ 

I 

I 



Appendices 

APPENDIX A: Histograms Displaying the Distribution of Index Scores for the Nine 



Components Areas 42 

APPENDIX B: The Pennsylvania Student Assistance Program Components and 

Indicators Handbook 59 

List of Tables 

TABLE 1 : Commonwealth Approved SAP Trainer Survey Participants by Region 14 

TABLE 2: Focus Groups By Site and Participants 15 

TABLE 3: Site Visits By Location and School Type 15 

TABLE 4: Site Visit Questions 16 

TABLE 5: Commonwealth Approved Trainers Survey Results 17 

TABLE 6: Mapping of Items to Indicators 28 



List of Figures 

FIGURE 1 : Pearson Correlation Coefficients For All Pairs of SAP Components ... 34 



I 

I 

I 

I er|c 



ACKNOWLEDGEMENTS 



We want to acknowledge the ideas, support, energy, and time of the following individuals. 
Recognized is the support and leadership of Biagio Musto, retired Chief, Division of Student and 
Safe School Services, Pennsylvania Department of Education and Henry Sontheimer, former 
Senior Evaluation Analyst with the Pennsylvania Commission on Crime and Delinquency. 

At the University of Pittsburgh we thank Robert Faux, Judy Bowlus and Beth Marco. At St. 
Francis Institute for Psychiatric and Addiction Services, thanks go to Mike Evans, Chief 
Operating Officer, as well as Pat S. Woodke and Regina Coleman. Thanks also go to the staff of 
KIT Solutions, Inc. (formerly XYZ Research Associates) for their technical support and data 
analysis, in particular Rebecca Moran and Kathy Scott. 

We received support and help from individuals at the Pennsylvania Departments of Education, 
Health and Public Welfare and the Commission on Crime and Delinquency. Within the 
Department of Education the following individuals are recognized: Maritza Robert, PhD, Bureau 
of Safe Schools and Student Services; John Weiss, Division Chief. 

The Commonwealth of Pennsylvania Network for Student Assistance Services Regional are 
thanked for their willingness to coordinate the regional focus groups. Thanks go to Marlene 
Kanuck, Kim Swamer, Dan Iser, Sheldon Winnick, Renee Urick, Marie Bozelli, Beth Sprentz, 
Deborah Strayer, Pam Kolega and Margaret Innes. Bemie Hoffman, Chief consultant to the 
Pennsylvania Department of Education for at-risk issues, is thanked for his contributions. 

Gene Boyle, Director of the Bureau of Drug and Alcohol Programs at the Department of Health, 
Joe Villone, Prevention Coordinator and Jose Pagan, SAP Specialist are recognized for their 
support. In the Department of Public Welfare Charles Curie, Deputy Secretary for Mental Health 
and Sherry Peters in the Office of Mental Health and Substance Abuse, are recognized. 

C.E. ’Dutch' Krauss of Ksys Inc. is thanked for his assistance with the Pennsylvania Student 
Assistance Program Statistical database and report. 



Appreciation goes to the professionals who helped with the PASAP focus groups at the annual 
conference, in particular Bob Wilson. We thank the evaluation advisory board members for their 
help, feedback and support: Joe Villone, Jose Pagan, Lee Ann Labecki, John Weiss, Marlene 
Kanuck, Sherry Peters, Robert Wilson, Henry Sontheimer, Ruth Williams and Doug Hoffman. 
Without their efforts, this evaluation would not have been possible. 



iv 



V U 



o 



9 



SECTION ONE 



INTRODUCTION 



Background 

In July of 1998, the Pennsylvania Commission on Crime and Delinquency (PCCD) funded an 
initiative to evaluate Student Assistance Programs (SAP) in Pennsylvania. The primary goal of 
the evaluation was to determine the overall efficacy of Student Assistance Programs in 
Pennsylvania. 



This evaluation investigated six areas of student assistance programs and developed a framework 
to guide future evaluations. The six areas were: 

• SAP referral processes 

• Parent consent practices 

• Intervention services 

• Contract provider assessments 

• School-based probation officers’ SAP involvement 

• SAP improvement and satisfaction 

Summary of Year One Results 

The conclusions of the first year of study as outlined in the Evaluation Report are as follows: 

• A defined well-communicated referral process is operational in most of the teams, and 
the potential referrers know how to access and utilize the student assistance team. 

• SAP teams contact and involve parents in the process. In cases where parents refuse SAP 
services, the primary reasons include denial that a problem exists and privacy issues. 

• SAP teams are making recommendations for a variety of services including assessments 
and are monitoring students on an ongoing basis. 

• D&A and MH liaison services are in place providing preliminary assessment and linkage 
with service providers. 

• An increasing number of schools are receiving services from school-based probation 
officers. Their involvement in SAP is evolving in accordance with the changing times in 
school settings. These officers have specialized knowledge of a limited student 
population and initially their participation was limited to their professional role and 
function. Currently their role is expanding to become a skilled resource to the team. 

• While currently not extensively practiced, the assessment of parent and student 
satisfaction has the potential to contribute to the continued quality improvement of SAP. 

• Suggestions for improvement of SAP were provided by team members and county 
administrators. However, their prioritization of suggestions reflects their role in the SAP 
process. 
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Aims of Year Two 



In July of 1999, the PCCD and PA Department of Education (PDE) funded a continuation of the 
initiative in order to identify essential components related to SAP, delineate indicators for each 
component area, identify factors related to exemplary and struggling SAP teams and outline a 
benchmarking process for technology transfer and continual improvement of SAP. 

The activities accomplished in the second year of the study include: 

• Literature review of SAP and adolescent risk factors 

• Identification of essential components of SAP based on literature review and Year 1 
results 

• Delineation of specific indicators of effective SAPs via focus groups, site visits and 
analysis of Year 1 data 

• Site visits of exemplary and struggling teams 

• Establishment of specific components and indicators 

• Development of a common unit of analysis to link data sources 

• Development of user friendly written materials explaining the utilization of benchmarks 
and indicators 

• Development and delivery of a training module for the Commonwealth Approved 
Training System 

The continuation of the SAP evaluation was accomplished by an interdisciplinary team of 
researchers and evaluators from three institutions: the University of Pittsburgh, School of 
Education; St. Francis Medical Center, Division of Psychiatric and Addiction Services; and KIT 
Solutions, Inc. 



SECTION TWO 



METHODOLOGY FOR DEVELOPING COMPONENTS AND INDICATORS 

The methodology for the component and indicator development was a modified Delphi process 
that consisted of a literature review, a survey and focus group process with the State-wide student 
assistance advisory committee, SAP Commonwealth approved trainers, Pennsylvania Association 
of Student Assistance Professionals conference participants, and members of county coordination 
teams members and SAP district councils, and school site visits. Presented in this section are the 
results of the literature review, descriptions of the survey and focus group process, and site visit 
protocol. 



Literature Review 

The first step in the development of the components and indicators was completion of a 
comprehensive literature review of research on school-based and school-linked programs and 
services that address barriers to learning for youth at risk for social and emotional problems, drug 
and alcohol use and abuse, and depression. Reviewed was literature from the fields of student 
assistance programs, health education, school psychology, guidance counseling, school health 
services, social work, school effectiveness, mental health counseling, drug and alcohol 
prevention, intervention, and treatment, and student outcomes and achievement. Guiding the 
literature review were three Pennsylvania Department of Education Basic Education Circulars (24 
PS: §15-1547, 1997; 22 PA:§12-31, 1997; 20 USC: 1232h, 1997) that focus on the structure and 
process of SAP in Pennsylvania, parental rights and role in SAP, and protection of pupil rights. 
Additional documents included the Commonwealth of Pennsylvania Department of Education 
Guidelines for the Commonwealth Training System (1998) as well as the Standards and Core 
Competencies for Student Assistance Core Team Professionals (1998). Also guiding the review 
were the Pennsylvania Department of Public Welfare, Office of Mental Health and Substance 
Abuse Services, Student Assistance Programs, Minimum Guidelines for County Mental Health 
Programs (1997) and the National Association of Student Assistance Program Guide lin es for SAP 
design and implementation (1999). From these sources nine component areas were identified as 
critical for effective student assistance program design, implementation, and evaluation in 
Pennsylvania. 

• Policy and Procedures 

• Communications 

• Referral Mechanisms 

• Parent Participation 
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• Team Planning 

• Intervention and Recommendations 

• Follow-up and Support 

• Training 

• Outcome Indicators and Evaluation 

The following review of the literature was undertaken to establish the practical and theoretical 
bases of the components and potential indicators of SAP effectiveness. The review embraced 
literature from the above mentioned to better understand how these components and potential 
indicators have been employed by others, thus informing SAP stakeholders of the variety of ways 
in which these components and indicators can inform and improve SAP practice. 

Policy and Procedures 

In order for any program to succeed clear statements of policy and procedure are required; 
furthermore, policies and procedures must be cohesive. Lacking cohesiveness often results in 
organizations working at cross-purposes. In SAP, clear and cohesive statements of policies and 
procedures are necessary if programs aimed at addressing barriers to student learning are to be 
successful (Adelman & Taylor, 1999). As Adelman and Taylor caution, fragmentation of policies 
and procedures concerned with overcoming barriers to learning only proves to be harmful to 
children, families, and neighborhoods. 

Those programs that are most effective in tackling the issues of children and families at risk are 
those that have adopted a community-based, holistic philosophy (Children, Youth, and Families 
at Risk, {CYFAR}, 1999). Children are seen in the contexts of family and community. The 
family serves as the entity that offers nurturance to children. The community offers, ideally, 
economic, educational, and health support to families. Moreover, effective programs are able to 
enlist the assistance of a variety of organizations, agencies, and citizens who work collaboratively 
in aiding children and families at risk. 

It is incumbent not only upon schools, but families, health care workers, the media, and 
community organizations to be actively involved in tackling health and social problems (Centers 
for Disease Control, {CDC}, 2000). Schools can help to provide cohesion to policies and 
procedures by providing a critical facility in which families, agencies, and community 
organizations are able to align their services with the needs of students. 

The CDC (2000) has developed an eight-component model of coordinated school health 
programs. While all eight of the components are critical to a coordinated school health program, 
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two of the components are especially relevant to the SAP model: Counseling and Psychological 
Services and Parent/Community Involvement. 

Counseling and Psychological Services entail group and individual assessment, intervention, and 
referral to enhance the emotional, mental, and social lives of students. Parent/ Community 
Involvement concerns the involvement of a broad-based constituency that includes parents, 
school health advisory councils, coalitions, and community resources to address the health needs 
of all students. Parents and community agencies, working together can help schools in a variety 
of activities directed at helping students overcome barriers to learning. Consistent parental 
involvement in all phases of the student assistance process is not only crucial to the success of all 
student assistance services, but assured (Commonwealth of Pennsylvania Basic Education 
Circular, 24§ 15-1547, 1998; National Association of Student Assistance Programs, {NASAP}, 
1999). 

NASAP (1999) as well as the Commonwealth Department of Education in the Basic Education 
Circular, 24§ 15-1547 (1998), place a great deal of emphasis upon program awareness. Such 
awareness programs involve the education of parents and the larger community concerning 
school policy on alcohol, drags, weapons, and violence, and the SAP services available to 
students. School Board Policy is essential in this process, for it clearly states a school's stance 
concerning drags and violence, the consequences of violations, and states the procedures for 
seeking help through SAP. School Board Policy also includes information concerning parents' 
rights and issues of confidentiality for students under 18 years of age. Policies and procedures 
that are revised regularly as laws relating to drag and alcohol use, parental involvement, violence 
and weapons change facilitate the creation of limits that help create a safe and secure 
environment so that all students can succeed. 



Communications 

Critical to the effectiveness of SAP is the success of efforts to communicate what SAP has to 
offer to all interested parties (students, teachers, administrators, and parents). The success of SAP 
depends on how well those who can benefit most know what it has to offer them. Adult 
collaboration is important in the development of a community of caring and to insure that all 
those involved grow from the process (Woodruff, Shannon, & Efimba, 1998). 




A goal of SAP awareness is to involve students, parents, agencies, and community members of 
efforts to develop safe and drag-free schools and communities (Commonwealth Department of 
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Education Basic Education Circular, 24§ 15-1547, 1998; NASAP, 1999). To attain this goal, 
parents and community members are informed of school policy concerning substance abuse and 
violence and what SAP services are available to students to help them overcome barriers to 
learning. Various methods present themselves for dissemination of SAP information. Each of 
these methods is especially suited to reach the various interested parties. The methods include: 
homeroom presentations, SAP posters, newsletters, parent education classes and training, 
coverage by the media (newspaper, radio, and television), and inclusion in parent and student 
handbooks (DiRienzo, 1990; NASAP, 1999). 

The CDC (2000) has stressed the need for parent and community involvement for enhancing the 
well being of students. A broad-based constituency of parents, agencies, and community 
members can offer support for school health programs. One of the ways in which parents are 
brought into the SAP process is by informing them that they will be notified if concerns are raised 
around their child's safety, health, or education and by assuring them that they will be involved in 
all aspects of the SAP process. Indeed. As Adelman and Taylor (1999) urge, efforts to address 
barriers must include a respect of all those involved and partnership in decision-making and 
governance. 

Referral Mechanisms 

When concern about a student arises, one of the most important roles an individual can play is 
assisting that student in seeking the help she or he needs (School Mental Health Project-UCLA, 
{SMHP}, 2000). Students infrequently seek help for mental health problems (Stiffman, Chen, 
Elze, Dore, & Cheng, 1997). Thus, it is critical that individuals be available who can identify 
students' problems and be knowledgeable about services. Stiffman et al. have identified four 
"service sectors" that may function as conduits to mental health services: child welfare, juvenile 
justice, health, and education. Stiffman et al. identify teachers as the professionals most likely to 
be consulted. 

It is critical that the nature and scope of the students' problems be appreciated from various 
perspectives, such as family, school staff, peers, and so forth. This also includes understanding 
the nature of the student's problem. The CDC (2000) suggests that psychological services be 
aimed at the mental, emotional, and social health of all students. The purpose of referral in SAP 
is the identification and referral of students confronting barriers to learning to a multi-disciplinary 
team (Commonwealth Department of Education Basic Education Circular, 24§15-1547, 1998; 



NASAP, 1999). Referrals may be made by parents, school administrators, peers, staff, or self- 
referral. 



According to SMHP (2000) a referral process that is client oriented should include: 

• Readily accessible information about resources 

• Mechanisms for helping students and their families appreciate the need for and 
benefits from resources 

• Accountability for problems of access (for example, cost and location) 

• Services for helping students and their families review their options and make the 
best decisions 

• Support and guidance to enable students and their families to connect with resources 

• Follow-up procedures to determine if referral decisions were appropriate and 
beneficial 

"Referrals are relatively easy to make; appropriate referrals are harder; and ensuring follow- 
through is the most difficult thing of all" (SMHP, 2000, p. 1). Case monitoring is essential for 
any referral to be effective. However, this is often overlooked, leaving stakeholders uninformed 
as to the status of the student’s care. Monitoring must continue until the student’s needs have 
been addressed. 



In order for school-based services to be effective in ameliorating the difficulties of students, it is 
essential that the efforts of all be coordinated (Larson, 1992). As more people become involved 
in efforts to help a student, information about that student will often need to be shared. At this 
point, confidentiality of information about a student and her or his family becomes a critical 
issue. In Jaffee v. Redmond, 1996, the United States Supreme Court made patent the importance 
of confidentiality in relation to mental health treatment. 
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Parent Participation 

The CDC (2000) urges schools to actively seek parent participation in any integrated approach for 
enhancing the well being of students. Educators can leam from the wisdom and experience of 
parents. Students can benefit from consistent goals at home and at school. Parents can leam 
skills for behavior management and academic support. For these things to happen parents need to 
be seen as partners and advocates (Vosler-Hunter & Hanson, 1992). As partners and advocates, 
parents need to prepare for their expanded role in their children’s education. First, as Vosler- 
Hunter and Hanson suggest, parents need to understand their rights. This should include 
encouragement from school staff and administration in order for parents to feel as though they are 
respected partners in their children’s education. Second, parents need to know that their 
involvement need not be limited to meetings, conferences with teachers, and home support of 
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their children, but may include membership on advisory boards. Parents may also be policy 
makers. To be effective policy makers parents need to rely on their expertise as parents, gamer 
support from other parents, be willing to take chances, and maintain their objectivity. 

Numerous research studies have demonstrated that parent and family involvement increases 
student achievement and success (Catsambis & Garland, 1997; Cotton, 1994; Epstein, 1991; 
Epstein, & Dauber, 1995; Griffith, 1997; Henderson, & Berla, 1994; Hoover-Dempsey, & 
Sandler, 1997; Metropolitan Life Survey of the American Teacher: 1998; Olmstead, & Rubin, 
1982; Paulson, 1994; Sanders, 2000; USED, 1997). The benefits of involving parents are not 
confined to the elementary grades. There are significant gains at all ages and grade levels. Junior 
and senior high students whose parents remain involved make better transitions, maintain the 
quality of their work, and develop realistic plans for the future. 

Children from diverse backgrounds tend to do better when parents and professionals collaborate 
to bridge the gap between the culture at home and the learning institution. Student behaviors 
such as alcohol use, violence and antisocial behavior decrease as parental involvement increases 
(Epstein et.al, 1995). 

A number of programs have been developed that encourage parent participation. Lewis, Marine, 
and Van Horn (1992) have compiled a list of guidelines that should be considered when 
developing a parent participation program: (a) The program should promote effective two-way 
communication and collaboration between parents and teachers; (b) The program should teach 
parents to actively support their children’s learning in collaboration with the school staff; (c) The 
school staff should provide ongoing support for parents; (d) Parents should get involved in 
program improvement efforts; (e) Ongoing parent training should be offered. 

It is a commonplace that parents are busy people. Effective programs need to accommodate the 
life styles of parents. Some existing programs address specific needs, with others address a 
variety of concerns. Whether programs address specific or general needs, it is critical that they 
increase parent participation in their children’s education. For example, Administrators of the 
South Bend, Indiana, Community Schools, sought to establish communication among parents, 
teachers, and administrators. In 1982, they created a parent network within the district that (a) 
provided an avenue for communications between home and school, (b) identified the needs and 
concerns of parents concerning their child’s education, (c) generated positive public support for 
the schools, and (d) identified community resources for both home and school (Lewis, Marine, & 



Van Horn, 1991). All of those involved in this program have found it beneficial in main taining 
communication between home and school, and has helped to establish understanding of the goals 
parents and educators have for children. Based upon this mutual understanding, parents and 
educators can support each other in attaining the best outcomes for children. 



However, for any school-linked service to be successful it may be necessary for agencies and 
schools to share information about individual students, for example, medical or academic records, 
or family history. That such information be kept confidential is essential. The right to privacy 
has been applied for two purposes: (a) to ensure that independence is maintained in making 
decisions and (b) to guarantee that personal information is kept private. 



Soler and Peters (1993) have described several reasons for protecting the privacy of children and 
families: 

• Confidentiality restrictions prevent the disclosure of potentially embarrassing 
personal information. 

• Confidentiality provisions also prevent the improper dissemination of information 
about children and families that might increase the likelihood of discrimination 
against them. 

• Protecting confidential information can be necessary to protect personal and family 
security. 

• Children and families want to avoid prejudice or differential treatment by people 
such as teachers, school administrators, and service providers. 

• Confidentiality provisions may also be necessary to encourage individuals to avail 
themselves of services. 



The Pennsylvania Department of Education Basic Education Circular 24§ 15-1547, 1998 assures 
parent rights to be involved in all phases of the process and see all school files pertaining to their 
child. The document further outlines: “Involvement of parents in all phases of the SAP process 
underscores parents’ role and responsibility in the decision-making process and is the key to 
successful resolution of problems (p.2, 1998).” 

Team Planning 

The role of the SAP team is to assist in gathering and evaluating objective data and faculty 
observations and to recommend school and community resources. This includes matching the 
needs of students with the most appropriate school and community resources (Commonwealth 
Department of Education Basic Education Circular, 24§15-1547, 1998). The over-arching goal is 
to reach as many students as possible and link them to those individuals and agencies best suited 
to help them (NASAP, 1999). 
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Team planning and organization is critical to SAP. Regular meeting times and places serve to 
make the team work effectively (DiRienzo, 1990;NASAP, 1999). Furthermore, the roles taken 
by team members need to be delineated, with each member of the SAP team acquiring a clear 
understanding of her or his roles and responsibilities (e.g. leader, secretary, case manager). 
DiCecco, Rosenblum, Taylor, and Adelman (2000) argue that a key to team planning is the 
clarification of roles and functions. In order for a team to be effective and to maintain operational 
mechanisms, DiCecco et al. have described three steps needed to ensure the effective 
development of a team: 

• Initiation of the process: Members of the school community are identified as 
prospective team members; the school agrees to maintain the team once it has been 
formed. 

• Development of mechanisms for maintenance: The team and its operations are 
institutionalized. This is necessary for the team to be maintained and to evolve. 

• Ongoing support provided: Once a team has been formed and working, it is critical 
that it be monitored and supported to insure its effectiveness. 

For a SAP team to function properly it needs an organization structure that clearly defines what is 
expected of team members. Furthermore, the team must be a part of the school insofar as it has a 
meets regularly and has access to resources (e. g. telephones, materials, and secretarial help). 
Communication among team members is essential as well (NASAP, 1999). 

Intervention and Recommendations 

Interventions addressing barriers to learning need to be comprehensive, multifaceted, and 
integrated (Adelman & Taylor, 1999). For instance, community agencies and in-school 
counselors can work together to help students with assessment, evaluation, and counseling for a 
variety of educational, substance abuse, mental health, or family concerns (Commonwealth 
Department of Education Standards & Core Competencies for Student Assistance Core Team 
Professionals, 1998; NASAP, 1999). Community agencies, including parents and law 
enforcement, can work together helping schools with technical assistance and prevention 
activities. 

As noted by Adelman and Taylor (1999), a multifaceted approach to intervention is critical. 
NASAP (1999) echoes this notion by urging a comprehensive multi-disciplinary approach to 
prevention, intervention, and support that is tied to other school-based programs. The 
Pennsylvania Standards and Core Competencies for Student Assistance Core Team Professionals 
urge also that SAP be part of the larger system of supports in school and in the community. 



Further, these competencies guide the team in a risk and protective factor framework that build on 
the student’s strengths and assets, as well as identifying the challenges and barriers to learning 
(1998). Integral to this effort is the identification and mobilization of services and personnel 
best suited to meet the needs of students. Also, consideration must be given to the diversity 
found among students, with actions taken based upon the needs of diverse populations. 

In order for a program to evolve and to best meet the needs of students apposite evaluation and 
accountability procedures are required in order to enhance intervention strategies, refine 
objectives, and improve program activities, making them more relevant to students' needs 
(Adelman & Taylor, 1999; Commonwealth Department of Education Standards & Core 
Competencies for Student Assistance Core Team Professionals, 1998; NASAP, 1999). 

Follow-up and Support 

Once an intervention has taken place and recommendations made it is critical to the process that 
follow-up and support services be in place to ensure that students receive the help they need 
(Commonwealth Department of Education Standards & Core Competencies for Student 
Assistance Core Team Professionals, 1998). Case management provides an approach to monitor 
each student's progress in academics and behavior and to identify needed changes in the services 
offered to the student (NASAP, 2000). Different situations will require different amounts of 
follow-up and support (Ideas into Practice, 1997). Thus, it is important that a system be in place 
that will assign case monitors based on the situation. 

As noted by Adelman and Taylor (1999), any program aimed at helping students overcome 
barriers to learning must be held accountable through quality improvement evaluations that are 
tied to performance-based evaluations. Continued problem analysis will help to determine if an 
intervention is appropriate and if the student is making progress (Ideas into Practice, 1997). A 
key to follow-up activities is the sharing of status updates among all parties involved. 

Training 

Critical to the success of Pennsylvania SAP is a well-trained staff, both in the school and in the 
community (Commonwealth Department of Education Basic Education Circular, 24§ 15-1547, 
1998). A reason a program may not succeed is inadequate training of staff (Kazdin, 1993). 
Thorough training of teachers and school staff is critical, according to Kazdin. Effective mental 
health services in schools require understanding of core competencies by all parties involved 
(Flaherty, Garrison, Waxman, Uris, Keys, Glass-Siegel, & Weist, 1998). Flaherty et al. stress the 



0 




11 



20 



importance of collaboration and point to the barriers that prevent collaboration. Coordinating 
these perspectives is critical. Flaherty et al. suggest that training in the collaborative process 
should be provided for all. Pennsylvania provides this through a comprehensive training system 
of private providers approved by the Department of Education (Commonwealth of Pennsylvania 
Department of Education Guidelines for the CATS (1998). Because each of these groups 
approach school services from different perspective, the Pennsylvania SAP training allows each 
group to recognize their own and others’ strengths and competencies, while, building a common 
knowledge base. 

The Pennsylvania Standards and Core Competencies for SAP Core Team Training describe the 
knowledge and skills that are considered necessary for effective SAP practice. Professional 
training for team members is required to ensure the suitability of the suggested services, and 
effective collaboration and fulfillment of state and federal laws protecting the privacy rights of 
parents and students. The thorough training of team members, which results in a certificate from 
the approved training provider, assures the board of school directors, school administrators, 
parents, students and the public that team members have received state of the art professional 
training consistent with accountable standards and proper professional actions (Commonwealth 
Department of Education Basic Education Circular, 24§ 15-1547, 1998). 

Such training must consider the unique culture of the school and family community 
(Commonwealth Department of Education Standards & Core Competencies for Student 
Assistance Core Team Professionals, 1998; NASAP, 2000). As suggested by Billy, Grady, 
Wenzlow, Brener, Collins, and Kann (2000) a major factor in school provision of health services 
"is the opportunity structure created by school and community characteristics" (p. 13). The 
school environment plays an integral role in the quality of a child's academic, social/emotional, 
moral, and behavioral functioning (Roeser, 1998). Adelman and Taylor (1998) argue for "a well- 
designed and supported infrastructure. . .to establish, maintain, and evolve, this type of 
comprehensive, programmatic approach" that will support school-wide enabling (p. 148). 
Enabling refers to policy-oriented school-wide efforts to address barriers to learning. 

Outcome Indicators and Evaluation 

The United States Department of Education’s Principles of Effectiveness requires schools to 
demonstrate positive outcomes for students as a result of programs, such as Student Assistance, 
which are funded through the Safe and Drug Free Schools and Communities Act. The need for 



more focused indicators and outcomes has created a need for evaluation that helps improve 
practice at the local level. 

It has been suggested that the use of administrative data is an effective and efficient way to 
identify students at-risk for excessive school absence and to define health strategies for school 
aged children (Kozinetz, 1995). Furthermore, Kozinetz argues that the use of such data may give 
clues and test hypotheses about the etiology of an outcome. Kozinetz suggests that excessive 
absences by a student are likely due to health-related reasons. With the identification of such 
students, the health-related factors that are likely contributing to the absences can be addressed. 

A focus on outcomes for students and parents has shifted the focus of interventions from 
activities to the improvements that are achieved (Wagner, 1995). In an attempt to better 
understand outcomes, both social and academic, for students with serious emotional disturbances 
(SED), Wagner utilized the longitudinal database compiled by the National Longitudinal 
Transition Study of Special Education Students (NLTS). Wagner found that students with 
different disabilities had different outcomes not only in comparison with their peers but also with 
students in the general population. Outcomes for SED students are especially disturbing in that 
they often find little success in school and in adult life. 

However, utilizing the NLTS database Wagner (1995) uncovered various factors related to 
improved outcomes for these students. These factors include parental involvement, expansion of 
curricular options to include vocational education, academic placement options, social 
integration, and collaboration among school, mental health and social service agencies. 

In a study of social competence promotion among inner-city and suburban adolescents, Caplan, 
Weissberg, Grober, Sivo, Grady, and Jacoby (1992) found that school-based social competence 
skill training was effective in improving coping skills and increasing social adjustment and 
interpersonal effectiveness. 

Literature Review Summary 

The literature review generated a list of 94 indicators divided among the nine component areas. 
Table 5 in SECTION THREE (p. 17) shows the complete list of indicators. The next step in the 
modified Delphi process was to receive feedback from key informants in the SAP field on what 
they considered the critical indicators of an effective SAP at the building level. An instrument 
listing all 94 indicators was developed. On the instrument, individuals are asked to rate each 
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indicator as being ’absolutely essential’, helpful but not essential’, or hot necessary’. 

Demographic information was collected on each respondent. 

Statewide Student Assistance Advisory Committee 
The initial draft of the 94 indicators was shared with the Statewide SAP advisory committee on 
January 24,2000 at a meeting in Harrisburg. Attending the meeting were 55 representatives from 
across the state. Participants included staff from the Departments of Education, Health, and 
Public Welfare, school districts, and community organizations. Working in small groups 
indicators were reviewed for each component area. The purpose of the review was to check the 
face validity of the instrument’s structure and content. Indicators were modified and refined 
based on the committee feedback. 

Survey of SAP Commonwealth Approved Trainers 
The first group surveyed with the component and indicator instrument was the SAP 
Commonwealth Approved Trainers. The instrument with a self-addressed stamped envelope was 
sent January 31, 2000 to all Commonwealth Approved Trainers (n = 86). The response rate was 
100%. Table 1 shows the responses by region. Twenty respondents did not identify their region. 
Seventy seven percent of the trainers reported being current and active SAP team members. 

TABLE 1 Commonwealth Approved Trainer Survey Participants by Region (n = 86) 



Region I 


4 


Region VI 


3 


Region II 


5 


Region VII 


22 


Region III 


9 


Region Vm 


20 


Region V 


3 


Region not identified 


20 



Focus Groups 

Five focus groups were conducted. The basis of the focus group was the component and 
indicator instrument. Members of the focus groups were asked to rate and discuss indicators on 
the scale of each being ’absolutely essential’, helpful but not essential’, or ’not necessary’. Three 
focus groups were conducted with county coordination teams and district councils. The county 
district council coordinators were contacted because the councils are comprised of a 
representative group of professionals from the field. The groups were with members of the 
Fayette County Student Assistance Program Coordination Team, Allegheny Intermediate Unit 3, 
and Armstrong/Indiana SAP District Council. Two additional focus groups were held at the 



annual conference of the Pennsylvania Student Assistance Professionals in Harrisburg. Morning 
and afternoon sessions were conducted. Table 2 summarizes dates, sites, and number of 
participants in the focus groups. 



TABLE 2 Focus Groups: Breakdown of focus groups by site and participants 



Date 


Site 


Participants 


February 28, 2000 


Fayette County 


16 


March 8, 2000 


Aium 


21 


March 16, 2000 


Armstrong/Indiana 


35 


March 27, 2000 


PASAP Conference~AM 


17 


March 27, 2000 


PASAP Conference-PM 


16 



School Site Visits: Exemplary and Struggling 
The final step in the modified Delphia process was to talk with school teams about the 
components and indicators. The site visits served as a method to check the reliability and validity 
of the indicators within the school and team setting. Visits were made to both exemplary and 
struggling school SAP sites. A nomination process was used to select school sites to visit. Each 
participant in the trainer survey and focus groups was asked to identify exemplary and/or 
struggling school SAP teams. These formed the basis for the school site visit selection process. 
Schools recommended for site visits were crosschecked to assure that the teams had participated 
in the State-wide team survey in year one as part of the year one evaluation process. A 
representative selection of schools was made by region, type, and grade level. The selection 
process was anonymous. Table 3 shows the site visit distribution. At no time during the school 
site visit process was a school identified as being exemplary or struggling. School visits were one 
half day. 



TABLE 3 Site Visits by location and school type 



Exemplary Schools 


Struggling Schools 


Region VIH - Suburban Middle School 


Region I - Urban Jr/Sr High School 


Region VIH - Urban High School 


Region II - Rural Middle School 


Region IX - Suburban Jr/Sr High 


Region VI - Rural High School 




SAP coordinators were contacted and sent a confirmation letter in preparation for the site visit. 
The component and indicator instrument was faxed to the SAP Coordinators who then made 
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copies to distribute to the team members. In addition to ranking the component qualities, teams 
were asked how they functioned in relation to the SAP components. Finally, a series of questions 
was used to clarify the reliability and validity of the indicators. Table 4 lists the questions. 

TABLE 4 Site visit questions 



• Which components are the most important to your program and why? 

• Which are the least important and why? 

• What specific indicators are contributing to your successes? 

• What specific indicators impede your success? 

• What components, if any, do you think your SAP team excels? 

• What are some of the challenges your team encounters? 

• What are your plans to improve your SAP program? 

• What would you build into a quality SAP program? 
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SECTION THREE 



RESULTS 



The following section presents the profile of findings from three sources during the modified 
Delphi process. In each phase of the process the component and indicator instrument was used to 
refine the list of indicators. The final list of components and indicators can be found in Appendix 
B. As part of each phase, the evaluation team met to analyze the data and refine the indicator list. 



Survey of SAP Commonwealth Approved Trainers 
The Commonwealth approved trainers survey results are shown in Table 5. The category of 
absolutely essential was the only rating considered in the data analysis. Subsequent to the data 
analysis the top five indicators for each component were used to shape and guide the focus groups 
and site visit processes. 

TABLE 5 Commonwealth approved trainers survey results (n=86) 

Component 1: Policy and Procedures 





l .% of total ' • 
responses 


Building administrator involvement and support 


94% 


Drug & Alcohol SAP liaison works with team 


86% 


Mental Health SAP liaison works with team 


83 % 


Confidentiality of SAP process complies with state and 
federal laws 


80% 


School crisis management plan 


71 % 


SAP coordinator for the building 


69% 


Policy violations and consequences for alcohol, drugs, 
violence, weapons, tobacco 


69% 


SAP structure and organization (including members and 
titles, clear delineation of roles & responsibilities, meeting 
times, membership selection criteria, member replacement 
process and leadership) 


69% 


Focus on removing barriers to learning and linking to school 
and community resources 


64% 


SAP is identified as school program consistent with 
Department of Education BEC 


62% 


Description of SAP services 


58% 


Annual report, and data summary procedures 


40% 
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Component 2: Communications 



1 


Indicator 


% of total 

! 

responses 




Staff in-service (type and frequency) - teachers, pupil 
personnel, support, administrative 


86% 




Faculty handbook contains SAP related information 


73% 




Student handbook contains SAP related information 


69% 




SAP program brochure that reaches parents 


64% 




SAP program brochure that reaches faculty 


64% 




Periodic review process for insuring that communications 
materials are accurate and up-to-date 


64% 




SAP program brochure that reaches students 


58% 




Student training (type and frequency: assembly, classroom, 
small group, individual) 


57 % 




Communications plan - specific SAP focus 


55 % 




Parent handbook contains SAP related information 


55 % 




Presentation to parents (type and frequency) 


47 % 


Component 3: Referral Mechanisms 




Indicator 


% of total 
responses 




SAP accessible to all targeted students 


87% 




Screening process - student data collection and review 
procedure 


80% 




Confidentiality guidelines for team well delineated 


80% 




Case monitoring (initial and ongoing) 


79% 




Formal referral process - written procedure, timeline, 
responsibilities and tasks 


78% 




Respect for and understanding of privacy rights - parents & 
students 


78% 




Consent process well delineated 


74% 




Release forms - consent to exchange confidential 
information with agencies 


71 % 




Decision making process 


67% 




Release forms - release records 


63% 
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Component 4: Parent Participation 



" •” ‘ Indicator ! 


% of total j 
responses 


Confidentiality guidelines 


/a 7 o 


Consent process 


74% 


Respect for privacy rights 


74% 


Release forms - consent to exchange confidential info. 


71 % 


Policy statement (parent/guardian as decision makers and 
active consent) 


66% 


Formal parent involvement procedure 


64% 


Mechanism for developing relationships with parents as 
students move through the SAP process 


58% 


Release forms - release records 


55% 


Mechanism for incorporating feedback into team process 


48% 


Mechanism for gathering parent input to inform and improve 
the overall process that the team is using 


45% 


Family education and information — prevention and general 
population focus 


28% 



Component5: Team Planning 



'in.';: Indicator . 


% of total 
responses ] 


Meeting time (regular meeting time sufficient to complete 
work) 


83% 


Case assignment and management procedure 


69% 


Regular meeting space with access to telephone 


67% 


Member role assignments and responsibilities articulated 


61 % 


Inter-team communication system 


52% 


Decision making procedure 


49% 


Inter-team conflict resolution strategy 


47% 


Meeting agenda 


43% 


Meeting minutes completed, reviewed, maintained 


40% 




28 

19 



Component 6: Intervention and Recommendations 



Indicator 


% of total ; 
responses 


Provide linkages as necessary to assist parents and students 
to access community services 


76% 


Case monitoring (student, parent, recommendation) 


76% 


Written guidelines for dealing with problems that are beyond 
the scope of the school’s responsibility - i.e., provision of 
treatment, suicidal assessment, etc. 


64% 


Assessments (school and community) monitoring and 
feedback to team 


63% 


Written information available on community resources and 
options 


56% 


Student and parent intervention and recommendation 
guidelines 


49% 


Mechanism to inform and improve school interventions and 
recommendations 


42% 


Tracking mechanism that measures outcomes 


37% 



Component 7: Follow-up and Support 



-indicator ' 


of total : j 
'responses 


Monitoring and follow-up of referred students 


72% 


Availability of in-school resources 


70% 


Availability of community resources 


69% 


Mechanism to promote student access to and compliance 
with in-school services 


66% 


Case monitoring functions (student, parent, intervention 
recommendation compliance) 


63 % 


Mechanism to promote student access to and compliance 
with treatment recommendations 


63% 


Referral to other resources in school such as guidance 
counselor, educational support groups 


61 % 


Mechanism to promote student access to and compliance 
with aftercare/continuing support 


56% 


Continuing care (aftercare) planning and follow-up 


54% 


Periodic SAP file review 


54% 


Mechanism to inform and improve follow-up and support 


48% 


Tracking mechanism for following student performance 


47 % 
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Component8: Training 



J'f: y-r. - Indicator 


% of total 
responses 


SAP team member training: basic/add-on level - PA state 
accredited provider 


88% 


SAP team maintenance - PA state accredited provider 


70% 


Adequate training schedule and budget 


66% 


SAP team member training: advanced level - PA state 
accredited provider 


57% 


SAP in-service training - school and agency staff - PA state 
accredited provider 


55% 


Materials and resources available and accessible 


50% 


SAP team member training: group facilitator training 


47% 


Annual training needs assessment targeted by groups (e.g., 
team members, parents) 


45% 


Mechanism to inform and improve training 


44% 



Component 9: Outcome Indicators and Evaluation 



-'T Indicator" 


% of total j 
responses 


Student data management system - school records: 
attendance, GPA, failed a course, retained, leaving 


69% 


Student monitoring: frequency, duration, intensity, services 
involvement 


67% 


SAP acceptability - stakeholders view program as useful, 
effective, and consistent (e.g., principal) 


66% 


Student data management system - behavioral outcomes, 
D&A, problem behavior 


61 % 


Student data management system - intervention and 
recommendation compliance 


55 % 


Improvement mechanism - student, team members, agency, 
school staff, and parents 


54% 


SAP efficacy - extent to which SAP meets goals of 
removing barriers to learning 


54% 


School monitoring of program 


47% 


Team monitoring: meeting minutes (members, students, 
recommendations) 


47 % 


Agency monitoring: referral patterns, level and extent of 
involvement, student outcomes 


47 % 


SAP team functioning process and outcome variables 
(interdisciplinary vs. multidisciplinary) 


41 % 


Pennsylvania Department of Education tracking system 


38% 
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Focus Group Results 



Policy and Procedures 

The overriding critical element to SAP policies and procedures was building administrator 
involvement and support. Having a building administrator who actively sits on the team was 
stated to be the key to long-term success. It was recommended that administrators understand the 
system well; as in some schools where they actually give referrals, use the forms, and follow 
students. Also identified as important was having a building coordinator (as opposed to district 
coordinator) who works with the liaisons so that no students fall through the cracks. Having a 
D&A/MH liaison who works with the team was important and particularly crucial for large 
districts. Having the SAP structure and organization clearly defined was also seen as a priority. 
As new people enter join the team, having clear policies and procedures assure that the process 
can continue uninterrupted. 

Communications 

Constant communication was seen as key to SAP functioning. Having a plan that addresses 
parents, faculty, and students were seen as essential. Maintaining consistent visibility by the use 
of pictures, posters, flyers, brochures, websites, and so forth, was seen as contributing to 
accessing SAP. Assemblies were seen as too impersonal and should not be relied on to 
communicate SAP, but could be used in conjunction with other means. It was recommended that 
every time parents come to school (e.g., for conferences, parent/teacher nights) it should be to 
educate them about SAP. Staff in-services and orientation were also seen as crucial to SAP 
success. In-servicing the school board was recommended. 

Referral Mechanisms 

Formal referrals were considered central to the SAP process, although there was discussion about 
what constituted "formal". Constant vigilance was stressed as a way to keep the referral process 
moving. For example, the home and school visitor hand delivers the letter if parent does not 
return it. Another school sends second letter by registered mail to assure it being received. 
Making SAP accessible to all targeted students was reported as important as was case monitoring. 

Parent Participation 

Having a mechanism for gathering parent input to inform and improve the overall team process 
was viewed as helpful. Although schools often did not have time for this, all felt it was a good 
idea. Formal parent involvement procedures and policy statement were viewed as essential. 
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Confidentiality guidelines, release forms (both consent to exchange information and release 
records), and respect for privacy rights were emphasized by the PASAP focus group. 

Team Planning 

Member role assignments and responsibilities need to be well articulated for team planning. A 
regular meeting time (one that fits schedule) and a regular meeting space with access to 
telephone, computer and email are crucial. Having a space conducive to discussion (e.g. around a 
table as opposed to a classroom) was also seen as helpful. Meeting minutes that are completed, 
reviewed, and maintained are needed to stay accountable. An inter-team communication strategy 
is also viewed as essential. "If we can’t resolve problems, how do we expect kids to?" quoted one 
focus group member. 

Intervention and Recommendations 

Fundamental to the intervention and recommendation process were assessments with the school 
and community. Monitoring and feedback to team was cited as being central to avoiding 
communication gaps with agencies/CYF. Having written information available on community 
resources and options and written guidelines for dealing with problems that are beyond the scope 
of SAP were seen as particularly helpful. Vigilant case monitoring is vital. 

Follow-up and Support 

Availability of in-school and community resources is required to facilitate the system. 

Monitoring and follow-up of referred students are crucial as often a team lets go of a student 
prematurely. Promoting student access to and compliance with both in and out of school services 
is essential. A periodic SAP file review helps to maintain this. 

Training 

SAP in-service training, both basic and advanced are necessary. Training twice a year is best as 
well as having advanced training on the latest updates. SAP Team maintenance is considered a 
priority. Team maintenance is described as also including team appreciation and thank-you’s for 
the additional work required by team members. Availability of substitutes for many different 
special programs is a constant problem for being on the team and being trained together. An 
adequate training schedule and budgets need to be included in the SAP plan. 
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Outcome Indicators and Evaluation 

Essential outcome indicators include SAP acceptability to stakeholders and the Pennsylvania 
Department of Education tracking system. Also was the very important question, "Did the child 
connect with a caring adult (during the SAP process)?" Furthermore, the evaluation system is the 
key to student monitoring. Focus group members were concerned about who was doing the 
monitoring (e.g., liaisons, coordinator, team members, principal). Furthermore, the SAP team 
must work to make time to do evaluation. The concern is that currently evaluation is a low 
priority. Plus it is hard to do. . 

Site Visit Results: Exemplary Schools 

Policy and Procedures 

Exemplary schools had very strong building and central office administrative support, either on or 
off the team. Administrators were highly visible and worked closely with the coordinator. 

Many also had actively involved drug and alcohol and/or Intermediate Unit SAP liaisons. A full- 
time coordinator, competent providers, and a therapist in the building assured continuity of care. 
Most teams reviewed their crisis management plans yearly to assure that SAP was thoroughly 
integrated into all policies and procedures. 

Communications 

Effective schools provided ongoing communication, not just at team meetings. Teams updated 
and revised their communication literature yearly disseminating this information widely through 
multiple sources. Constant and ongoing communication made SAP accessible to students, 
parents, and school personnel. 

Referral Mechanisms 

Schools trained all teachers, including special education teachers in the referral process. Having 
clear directions to make the referral process user-friendly was developed and constantly refined 
as needed. 

Parent Participation 

Parental involvement was a priority for exemplary programs even when contact was difficult. 
Schools persevered with home and school visitors, registered letters, etc. to assure compliance. 
Drug and alcohol and mental health treatment agencies often completed Parental Satisfaction 
Surveys. Family confidentiality was maintained. 
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Team Planning 

A time commitment, including preparation time, was included in the scheduling. Giving staff 
adequate time was crucial so that team responsibilities did not conflict with other priorities. 
Administrators, counselors, and liaisons attended meetings without fail. 

Intervention and Recommendations 

In-school mental health treatment for identified SAP students was encouraged. 

Exemplary schools had well-built agency networks particularly with hospital and partial 
programs. 

Follow-up and Support 

Continuous communication was developed to assure that students were adequately supported. 
Staff was kept informed and held accountable for follow-up. 

Training 

Even when teams had very small training budgets, teams regularly did a variety of maintenance 
and continuing education activities. They observed that actively participating in these activities 
helped them assess their work and make improvements. Regular training allowed them to 
function better, avoid bum out, and keep up to date. 

Outcome Indicators and Evaluation 

Good secretarial assistance allowed for continuous data management. Performance data was 
reviewed yearly and disseminated to stakeholders. Core teams were credited with successes and 
statistics and data were used to support this. Evaluation was valued and given adequate time to 
be incorporated into the process. 

Site Visit Results; Struggling Schools 

Policy and Procedures 

Insufficient and unenforced policies and procedures were common in struggling schools. Most of 
these schools teams were not familiar with their policies, and if they were they were ’on paper’ 
only. 

Communications 

Style, philosophical and differences in involvement between team members caused 
communication difficulties. Having no mechanism to resolve problems, teams were unable to 
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improve their functioning. Since trends change quickly, if the team is not communicating, team 
members will continue to get further behind. 

Referral Mechanisms 

There was a widespread belief that once students were identified and referred, their job is 
finished. It was rarely known if a student made it to the referral or not. Teachers wanted 
immediate feedback when a student was referred. This caused frustration and conflicting 
relationships among staff and team members. 

Parent Participation 

Schools often had a low opinion of parent’s interest or capacity to become involved. Truancy and 
attendance problems were common. With no school staff to assist with follow-through, there was 
little family involvement. 

Team Planning 

School schedules determined team planning. When teachers had extra periods, joint periods, or 
study halls duties, planning periods were lost. Often students were assigned without any meeting. 
When team members traded cases, students and parents were often lost in the process. 

Intervention and Recommendations 

When there was a lack of available community resources, team members were unable to do their 
jobs. Also, when there was constant liaison and staff turnover, the process stopped. In some 
cases there was a problematic relationship with the county mental health agencies because 
Managed Care had prevented families from receiving needed services. 

Follow-up and Support 

School teams were not informed of behavioral plans and/or aftercare instructions when a student 
was being released to a rehabilitation or psychiatric hospital. 

Training 

Staff does not willingly attend in-services. A lack of substitutes or scheduling problems 
interfered with training efforts. 

Outcome Indicators and Evaluation 

Evaluation is not well regarded and perceived as a chore. It was done haphazardly, if at all. 



SECTION FOUR 



INCORPORATION OF COMPONENTS AND INDICATORS RESULTS TO THE FIELD 

The development of the components and indicators offers the opportunity to assess SAP 
outcomes. Working toward that goal an investigation was undertaken in year two to link 
outcomes to SAP implementation at the school level. This section describes the investigation’s 
process and findings. 



Mapping the Data 

The development of meaningful components and indicators for SAP meant taking advantage of 
all possible sources of data. Therefore, the methodology was developed in such a way that the 
qualitative data collected from the current literature, site visits and focus groups could be linked 
with the quantitative data gained from the Pennsylvania Student Assistance Program State-Wide 
Survey and the Pennsylvania Department of Education’s 1997-1998 State Performance Report on 
the Student Assistance Program. 

Toward this end, a methodology for mapping the qualitative data onto the proposed indicators 
was developed. First, each survey item was examined to determine whether or not it provided 
evidence to support one or more of the indicators. Once this process was complete, it was 
replicated in the same manner for the items on the State Performance Report. The end result was 
a list of the nine components and, for each of the indicators, the instrument items that correspond 
to and/or give evidence in support of that indicator. 



The main goal to be achieved by the development of components and indicators of SAP team 
success is a self-evaluation tool that individual SAP teams can use to assess their own 
functioning. The mapping of individual items onto the proposed indicators served as the first step 
towards a general scoring measure, wherein the sum of the individual items can serve as a “score” 
that represents how well the team is achieving the standard set by the indicator. 




During the mapping process, several of the indicators identified by SAP team members as 
“absolutely essential” to effective SAP team functioning were found to have no related survey 
items. To eliminate this problem and further streamline the mapping process, the full list of 
indicators and corresponding items was examined. Where it was found that a particular item gave 

little support for an indicator, that item was either eliminated or moved to support a different 
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indicator. In addition, some items were given more weight in the scoring procedure; if one item 
supporting an indicator seemed to be a stronger and more precise representation of the overall 
indicator than other items (based on what was gleaned from the literature, obtained from focus 
groups, and the research team’s experience with SAP), that item was weighted so that it would 
constitute a greater portion of the overall indicator score. The final list of indicators and 
supporting items is provided in Table 6. 



TABLE 6 Mapping of Items to Indicators 

Component 1: Policy and Procedures 



Indicator 


Survey items Event items I 


1 . Building administrator involvement and support 


D, F, 2 




2. Drug & Alcohol and/or Mental Health SAP liaison 
works with team 


D, F, 2, 15, 15a, 16, 
16a 


- 


3. SAP coordinator for the building 


D, E, 2, 4, 5, 8a, 24 


- 


4. Policy violations and consequences for alcohol, 
drugs, violence, weapons, tobacco 


10, 13, 17 


- 


5. SAP structure and organization (including members 
and titles, clear delineation of roles & 
responsibilities, meeting times, membership 
selection criteria, member replacement process and 
leadership) 


C, E, 31 


- 


6. Focus on removing barriers to learning and linking 
to school and community resources 


9, 11, 13, 17, 29 


- 



Component 2: Communications 



I Indicator 


Survey items 1 


Event items" 


1 . description or sat services 


1 




2. Staff in-service (type and frequency) - teachers, 
pupil personnel, support, administrative 


1,31 


- 


3. Student communication strategy 


1,31 


- 


4. Parent communication strategy 


1,31 


- 




Items from the Pennsylvania Student Assistance Program State-Wide Survey 
2 Items from the Pennsylvania Department of Education’s 1997-1998 State Performance Report on the 
Student Assistance Program 
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Component 3: Referral Mechanisms 



Indicator j Survey items 


Event items as- 


1. SAP accessible to all targeted students ! 31 1 


2. Screening process - student data collection and 
review procedure 


8 | 


3. Confidentiality guidelines and consent process well | 
delineated; respect for and understanding of parents’ | 
& students’ privacy rights; use of release forms 
(consent to exchange confidential information with 
agencies and to release records) 


8b, 26 


- 


4. Formal referral and decision making process - 
written procedure, timeline, responsibilities and 
tasks 


3, 6,7 


- 



Component 4: Parent Participation 



Indicator 


Survey items 


Event items 


1 . Formal parent involvement procedure; mechanisms 
for developing relationships with parents as students 
move through the SAP process and for gathering 
parent input to inform and improve the overall 
process that the team is using 


22, 23. 26,31 


15 



Component 5: Team Planning 



Indicator 


Survey items 


Event items 


1 . Meeting time (regular meeting time sufficient to 
complete work) 


C 


- 


2. Member role assignments and responsibilities 
articulated 


j E, 4, 5, 8a, 24 


- 



Component 6: Intervention and Recommendations 



. •**£'•«?• Indicator • .n . 


Survey items 


Event items*: - 


1. Provide linkages as necessary to assist parents and 
students to access community services; written 
information available on community resources and 
options; written guidelines for dealing with 
problems that are beyond the scope of the school’s 
responsibility - i.e., provision of treatment, suicidal 
assessment, etc. 


11, 15, 15a, 16, 
16a, 17.31 

| 


18a, 18b. 18c 



Component?: Follow-up and Support 



Indicator | Survey items 


Event items 


1. Case monitoring: initial, ongoing, and follow-up of | 

referred students, and a mechanism to inform and ! 29 

improve follow-uo and suoport ! 


16, 17b, 18c, 
20 


2. Availability of resources (in-school and community) j 14, 15, 15a, 16, 

1 16a. 17. 21 


17b, 18b 


3. Mechanism to promote student access to and 
compliance with: (a) in-school services, (b) 
treatment recommendations, and (c) compliance 
with aftercare/continuing support 


31,32 


- 



Component8: Training 



Indicator 


Survey items 


Event items: . . 


1. SAP team member training: basic/add-on and 
advanced level - PA state accredited provider 


L, M 


- 


2. SAP team maintenance - PA state accredited 
provider 


31 


- 


3. Adequate training schedule and budget 


r 3i 


- 



Component 9: Outcome Indicators and Evaluation 



Indicator,:- 7'4. 


^ Survey items 


' Event iternsrii 


1 . Student data management system & tracking 
mechanisms 


14, 29, 29a 


17b, 18c, 19, 
20,21,22,23, 
24, 25 


2. Monitoring & improvement mechanisms 


r 28. 28a. 29. 29a 


- 



The next step in the development of components and indicators was the analysis of quantitative 
data to validate the proposed components and indicators and to establish (if possible) 
relationships between these indicators and team functioning. A description of the quantitative 
data and the analyses performed follows. 



Description of Databases from Year One Evaluation 
The first database consists of SAP team member responses to the Pennsylvania Student 

Assistance Program State-Wide Survey, which was conducted in January of 1999. The purpose of 

the survey was to obtain the information regarding the structure of the Student Assistance 

Program in the school, the effectiveness of the program and the challenges encountered from the 

perspective of the team member. Data collected from the survey was entered into a central 

database. 





The second data source utilizes the Pennsylvania Department of Education’s 1997-1998 State 
Performance Report on the Student Assistance Program. Data are gathered annually from SAP 
teams in public, non-public, and area vocational-technical schools regarding the services provided 
to students during the school year and are based on the team member’s perspectives in reviewing 
each student who has been seen by SAP during the school year. Aspects included in the report 
include referrals to the team, types of services recommended, types of assessments, team 
knowledge regarding student access of services and changes in performance indicators such as 
academic achievement, advancement, attendance, and student improvement. These data, which 
are gathered both objectively and subjectively by team members, are entered on scanning sheets 
and forwarded to a central location for processing. 

A limitation of the database is that it does not provide an objective comparison of change in 
student performance over time, which may be attributed to the student’s involvement in SAP. In 
other words, it does not include students’ pre- and post-SAP performance records. There were 
also some inconsistencies in the numbers reported which suggest that some of the team members 
may not have been completing the forms accurately. 

Procedure for linking two databases 

The planned analysis of both the SAP Survey data and the state-compiled data reported by team 
members was predicated on the ability to determine a common unit of analysis. Whereas the 
SAP survey results produced one set of responses per survey respondent, the data collected by the 
state represents the status of each SAP referral. Since these reports are only identified by case 
numbers, one student could conceivably have produced multiple reports, if that student was 
referred more than once in the same school year. This discrepancy in the unit of analysis for the 
two databases was reconciled through the development of a common unit of analysis. 

The SAP survey database contained a total of 1207 records, one record per SAP team member. 
The Statewide Performance Database contained a total of 59,269 records, one record per case (or 
event). One variable contained in both databases was an identifying code for a particular school 
building. Hence, the records in both databases could be categorized according to the school 
building from which the responses were obtained. This variable (school building) was used as the 
common unit of analysis for the construction of the combined database and subsequent analyses. 

Both survey and event data had been collected for a total of 159 school buildings. Due to missing 
data in both databases, only 147 school building codes were present in the two databases. Use of 
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a common unit of analysis required that the user- or event-level data be aggregated or 
manipulated to create a new set of variables at the school building level. This involved the 
development of an index for each variable present in the two original databases. For the data in 
the Statewide Performance Database, the index variable was most often a percentage of total 
referrals. As an example, the ninth item on the data collection form records the race of the 
referred student as White, Black, American Indian, and so forth. The building-level index 
variable simply represented the percentage of a school building’s referrals that were categorized 
in each of the race categories. Similarly, the outcome variables were aggregated to represent the 
percentage of referrals per school building showing improvement, and so forth, instead of the 
outcome for each individual referral. 

Data from the Statewide Survey were aggregated in a different manner, since each school 
building participating in the survey had a varying number of respondents (with respondents 
serving in different roles in the building’s SAP program). Additionally, SAP team members from 
the same school building did not necessarily respond unanimously to survey items. Therefore, 
the index for many of the survey items was constructed from the most frequently endorsed 
response among the respondents for each school building. For example, item C of the survey 
asks: 

How often does the SAP team meet during the school year? (MARK ONLY ONE) 

O Less than once a month 
O Once a month 
O Every 2 weeks 
O Once a week 
O Two or more times a week 
O Unsure 

A frequency distribution of team member responses to this item was examined; if the team 
members were in consensus, the value of the index variable for that school building was simply 
the unanimous response. If a consensus among respondents did not exist, but a clear majority of 
a team’s member selected a particular response, that response was used for the value of the index 
variable. As an example, if a team consisting of 6 members responded to item C such that 5 
members chose “Once a month” and 1 member chose “Every 2 weeks,” the most frequently 
chosen response, “Once a month,” was used for the index variable. 




ERIC 



32 



In cases where neither a consensus nor a clear majority existed, the responses were sorted by the 
respondent’s professional position in the school. Responses from building administrators and 
SAP coordinators were given more weight in “tie-breaking” situations. If additional information 
could not give rise to a clear majority, the more conservative alternative was used for the index 
variable. 

This methodology, applied consistently, resulted in the creation of a single database with school 
building as the unit of analysis. This database was then utilized to validate indicators for each of 
the nine component areas that contribute to SAP program success. 

Findings 

Specific Components and Indicators 

For each of the nine component areas, the corresponding indicators were “scored” using the 
quantitative data sources mentioned above. Therefore, each of the 147 school buildings obtained 
a score that represented how well their SAP team was meeting the standard set by each of the 23 
indicators. For ease of interpretation and consistency, each of the indicator scores was placed on 
the same scale. The final index for each indicator was a score on a scale from 0 to 100. Higher 
scores represent more evidence of attaining the standard set by the indicator; therefore, higher 
scores can be interpreted as evidence of successful functioning in that specific indicator domain. 
Histograms displaying the distribution of index scores for all nine component areas are shown in 
Appendix A. 

Specific Components and Indicators That Relate to Team Functioning 

The next step of the analyses focused on identifying relationships among the components and 
indicators and how the indicators may relate to team functioning. A matrix of Pearson correlation 
coefficients was prepared for the nine components. This matrix is displayed in Figure 1 . 
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FIGURE 1 Pearson Correlation Coefficients for All Pairs of SAP Components 
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Significant correlations (p < .05) are marked in bold. (Components: 1 = Policy and Procedures; 

2 = Communications; 3 = Referral Mechanisms; 4 = Parent Participation; 5 = Team Planning; 

6 = Intervention and Recommendations; 7 = Follow-up and Support; 8 = Training; 9 = Outcome Indicators 
and Evaluation) 

As shown by the correlation matrix, several of the components are related to each other in such a 
way that would have implications for SAP team functioning. Parent Participation (Component 4) 
was found to have significant correlation with five of the nine components. Likewise, Follow-up 
and Support (Component 7) was significantly correlated with five components. According to the 
findings, strengthening these two components would have the biggest positive impact on SAP 
practices. 



Strengthening parent participation, according to the findings would yield improvements in SAP 
implementation in the areas of Policy and Procedures (Component 1), Referral Mechanisms 
(Component 3), Team Planning (Component 5), Follow-up and Support (Component 7), and 
Outcome Indicators and Evaluation (Component 9). Likewise, improvements in Follow-up and 
Support will have a positive impact on Communications (Component 2), Referral Mechanisms 
(Component 3), Parent Participation (Component 4), Intervention and Recommendations 
(Component 6), and Training (Component 8). 



ERIC 



Furthermore, Team Planning (Component 5) and Training (Component 8) were found to have 
significant correlation with three of the eight components. Improvements in Team Planning will 
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affect Referral Mechanisms (Component 3), Parent Participation (Component 4) and Outcome 
Indicators and Evaluation (Component 9). Strengthening Training (Component 8) affects 
Referral (Component 3), Parent Participation (Component 4) and Follow-up and Support 
(Component 7) 

A final step was taken to investigate SAP outcomes in relation to the components and indicators. 
An attempt was made to explore the correlations between the components and indicators and the 
event databases. Specifically, the relationship of the proposed indicators to the student outcome 
variables collected for the state performance database was investigated. None of the nine 
components or the indicators was significantly correlated with the six outcome variables in the 
state performance database. These outcome variables measured (a) the percentage of each school 
building’s referred students who improved since the current year’s referral; (b) percentage of 
students with improved attendance; (c) percentage of students whose general academic 
performance improved; (d) percentage of students suspended since their referral; (e) percentage 
of students who violated D&A policy since their referral; and (f) the percentage of referred 
students who were promoted/graduated. 



Though none of these correlations were found to be statistically significant, the weakness of the 
outcome data (as mentioned in the description of the Statewide Performance Database, please see 
the year one report) may account for the results. With stronger, more valid outcome measures, it 
is possible that future analyses may discover potential predictive relationships between the 
proposed SAP components and indicators and student outcomes. 
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SECTION FIVE 



TECHNOLOGY TRANSFER 
Components and Indicators Handbook 

The final element of the SAP evaluation was to translate the findings into a user friendly and 
accessible package for the SAP practitioners. Motivating the material development was the need 
to focus on how to improve SAP practice with an emphasis on best practices. Therefore, the 
materials developed focused on supporting best SAP practice at a local level. The materials are 
primarily focused on school SAP teams. They have been packaged in a single booklet available 
from the Pennsylvania Department of Education and on the web (http://www.sap.state.pa.us/): 
Pennsylvania Student Assistance Component and Indicator Handbook. The handbook contents 
are contained in Appendix B. 

Central to the Handbook is the Pennsylvania Student Assistance Program Effectiveness Checklist 
and the Pennsylvania Student Assistance Program Performance Improvement Plan. The 
Checklist items are the component indicator items found through the evaluation to be measures of 
SAP effectiveness. The Improvement Plan uses data generated from the Checklist to guide the 
plan development. Use of the Checklist with specific stakeholder groups is also included (e.g. 
parents, administrators). 

How successful attempts are at bettering SAP at the school level depends on adequate 
implementation of improvement strategies presented in the handbook. A 360 degree process is 
described and encouraged as a means to collect information and feedback from the range of SAP 
stakeholders. Furthermore, schools are instructed on how to develop and use individual school 
SAP benchmarks of each component. 

The Commonwealth Approved Trainers and Department of Education staffs were trained in the 
use of the Handbook with SAP team members and stakeholders in August, 2000. Since the 
training the Handbook has been widely disseminated throughout Pennsylvania. Initial utilization 
is high with CAT trainers reporting dissemination through training programs, in-services, and 
individual school consultations. 




In the future, the components and indicators can serve as the structure for SAP evaluation and 
research. It provides the opportunity to compare and contrast SAP programs with the goal of 
improved SAP practice and services to students. 
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APPENDIX A 



Histograms Displaying the Distribution of Index Scores for the Nine Components Areas 
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Frequency Frequency 



Component 2: Communications 
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Frequency Frequency 



Parent communication strategy 
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Screening process 
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Formal referral and decision making process 



Frequency Frequency 



Component 4: Parent Involvement 




Component 4: Parent Involvement 



Formal parent involvement procedure 
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Frequency Frequency 
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Component 5: Team planning 



Std. Dev = 10.00 
Mean = 50.0 
N = 147.00 




Component 5: Team planning 





Meeting time 



Frequency Frequency 



Member role assignments/responsibilites 




40.0 45.0 50.0 55.0 60.0 65.0 70.0 



Member role assignments/responsibilites 



Component 6: Intervention & recommend 
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Std. Dev = 10.00 
Mean = 50.0 
N = 147.00 



Component 6: Intervention & recommendations 



Frequency Frequency 



Provide linkages to school and community 




Provide linkages to school and community resources 



Component 7: Follow-up and support 

100 
80 
60 
40 

20 
0 

20.0 30.0 40.0 50.0 60.0 70.0 

Component 7: Follow-up and support 






Std. Dev = 10.00 
Mean = 50.0 
N = 147.00 
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Frequency Frequency 



Availability of resources 




Mechanism to promote access and compli 

120 




Mechanism to promote access and compliance 
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Frequency Frequency 




Std. Dev = 10.00 
Mean = 50.0 
N = 147.00 



Components: Training 



SAP team member training 

100 



80-1 




SAP team member training 
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Frequency Frequency 



SAP team maintenance 



100 




SAP team maintenance 



Adequate training schedule and budget 




Adequate training schedule and budget 
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Frequency Frequency 



Component 9: Outcome indicators and ev 




Std. Dev = 10.00 
Mean = 50.0 
N = 147.00 



Component 9: Outcome indicators and evaluation 



Monitoring and improvement mechanisms 



Std. Dev = 10.00 
Mean = 50.0 
N = 147.00 

50.0 60.0 70.0 

Monitoring and improvement mechanisms 
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APPENDIX B 



The Pennsylvania Student Assistance Program Components and Indicators Handbook 
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The Pennsylvania Student Assistance Program 
Components and Indicators 

The Pennsylvania Student Assistance Program (SAP) ensures that students experiencing barriers 
to learning related to substance abuse and mental health problems are identified and linked to 
appropriate supportive services in the school and community. This booklet highlights the results 
of the Pennsylvania Student Assistance Evaluation Project. The project focused on the best SAP 
practices. Nine essential components of SAP practice were delineated: Policy and Procedures, 
Communications, Referral Mechanisms, Parent Participation, Team Planning, Intervention and 
Recommendations, Follow-up and Support, Training, Outcome Indicators and Evaluation. For 
each component area there are specific performance indicators. The component areas and then- 
corresponding performance indicators were derived from a review of SAP literature, key 
informant interviews, discussions and focus groups. A number of the performance indicators 
were validated through cross-analysis with existing SAP databases. The components have their 
roots in the Pennsylvania Student Assistance Program Model which is comprised of the following 
four phases. 



Pennsylvania Student Assistance Program Model 



Phase One involves referral. 

Phase Two involves team planning. 

Phase Three includes team interventions and recommendations. 
Phase Four involves follow-up and support. 



Referral 



Team Planning 



Intervention and 
Recommendation 



Support and 
Follow-up 




Identify problem behaviors 
Referral to core team 
Initial fact-finding 
Parent contact and participation 



Information gathering 
Student conference 
Team planning 
Parent conference 



Intervention - in-school or 
community resource 
Recommendation for services 



Support services for students/identify 

problem behaviors 

Referral to core team 

Initial fact-finding 

Parent contact and narticination 



The goal of the components and indicators is to support and involve all SAP stakeholders in 
continuous improvement of SAP. Presented in the booklet are the SAP Component Areas and 
Indicators, Component Indicator Checklist, Strategies for using the Checklist, and how to develop 
benchmarks for your SAP. 
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SAP Component Areas and Indicators 

(The indicators in bold have been validated through research) 

Component 1. Policy and Procedures 

• Building administrators are involved and support SAP 

• Drug and Alcohol and Mental Health SAP liaison works with team 

• SAP Coordinator for the Building 

• Policy violations and consequences for alcohol, drugs, involving weapons, tobacco 

• SAP Structure and organization (including members and titles, clear delineation of 
roles and responsibilities, meeting times, membership selection criteria, etc.) 

Component 2. Communications 

• Description of SAP Services for faculty, students and others including handbooks, 
brochures, etc. 

• In-services for teachers, pupil personnel, support staff, and administrators provide time 
and support for SAP informational updates 

• Specific student communication strategy 

• Specific parent communication strategy 

Component 3. Referral Mechanisms 

• SAP is accessible to all targeted students 

• Formal referral procedures and decision-making process 

• Screening process includes clear and consistent student data collection and review 
procedures 

• Confidentiality guidelines for team are well delineated with members demonstrating 
respect for and understanding of parents’ and student’s privacy rights 

• Cases are continuously monitored 

Component 4. Parent Participation 

• Formal parent involvement procedure 

• Policy statement defining parents/guardians as decision makers who provide active consent 
and includes a formal parent involvement and satisfaction procedure 

• Confidentiality guidelines known and respected 

• Demonstration of respect for parent and family privacy rights 

• Clear and consistent parent consent process and procedures 

• Information release form process and procedures for consent to exchange confidential student 
information 

Component 5. Team Planning 

• Regular meeting time sufficient to complete SAP work 

• Members’ role assignments and responsibilities articulated (e.g. leaders, secretary, case 
manager) 

• Case assignment and management procedures 

• Regular meeting space with access to telephone 

• Inter-team communication system 
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Component 6. Intervention and Recommendations 

• Support and provide linkages for students and parents to access school and comm uni ty 
services 

• Team monitors and receives feedback on school and community assessments 

• Continuous monitoring of student progress, parent involvement and recommendations 

• Written guidelines for dealing with problems that are beyond the scope of the school’s 
responsibility (e.g. provision of treatment, suicidal assessment) 

• Written information available on school and community resources, services, and options 

Component 7. Follow-up and Support 

• Procedures promote student access to and compliance with school and community 
services and treatment recommendations 

• School resources are available and accessible 

• Student follow-up procedure and process promotes student contact and support 

• Community resources are available and accessible 

• Continuous monitoring of student progress, parent involvement and treatment 
recommendations compliance 

Components. Training 

• Team members participate in all Commonwealth Approved Training 

• Team members participate in a maintenance and development tr aining program 

• Team has adequate training schedule and budget 

• Team members have opportunity and support for advanced SAP training 

• School and community agency staff participate in ongoing SAP training 

Component 9. Outcome Indicators and Evaluation 

• SAP monitoring and improvement mechanisms are in place 

• SAP team maintains a student data management system to track student attendance, GPA, 
failed courses, grade retention, and school leaving 

• Student SAP service participation and utilization system provides accurate and timely 
information 

• Student interventions, recommendations, and outcomes are regularly assessed for quality and 
goal attainment 

• Stakeholder input and suggestions are solicited and utilized 

• SAP satisfaction information is solicited and utilized 
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Pennsylvania Student Assistance Program 
Component Indicator Checklist 



Pur pose 

The Pennsylvania Student Assistance Program Component Indicator Checklist is designed to help 
you determine those elements of your SAP program in which your SAP program is the strongest 
and those in which your SAP program is in need development. Furthermore, through util izat ion 
of the various strategies discussed in this handbook, schools can develop their individual SAP 
benchmarks for each indicator. 

Importance 

Accurate diagnosis and feedback is an important first step in development. The Checklist 
feedback you receive will help you and your school benefit from your SAP program’s strengths 
and target developmental activities to address your SAP program’s specific developmental needs. 

Instructions for Completing the Checklist 

1. Be sure you fill in all of the boxes at the top of the next page. 

2. This checklist consists of 46 items that represent the essential SAP components. Read 
each item carefully and consider as it pertains to your SAP program. 

3. To the right of the items rate your SAP program’s current performance by circling the 
appropriate number. Complete the rating using the scale: 1= Never Present, 

2 = Sometimes Present, 3 = Present All Of The Time. 
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Pennsylvania Student Assistance Program 
Effectiveness Checklist 



Member name 


Date 


Building 


District 



Instructions: Please rate each SAP component indicator for your SAP by using the following scale: 

1: = Never Present 2: = Sometimes Present 3: = Present All Of The Time 



Component 1 Policy and Procedures 



Building administrators are involved and support SAP 


1 


2 


3 




Drug and Alcohol and Mental Health SAP liaison works with team 


1 


2 


3 


SAP Coordinator for the Building 


1 


2 


3 


Policy violations and consequences for alcohol, drugs, involving weapons, tobacco 


1 


2 


3 


SAP Structure and organization (including members and titles, clear delineation of 
roles and responsibilities, meeting times, membership selection criteria, etc.) 


1 


2 


3 


Total 







Component 2 Communications 



Description of SAP Services for faculty, students and others including handbooks, 
brochures, etc. 


i 


2 


3 




In-services for teachers, pupil personnel, support staff, and administrators provide 
time and support for SAP informational updates 


1 


2 


3 


Specific student communication strategy 


i 


2 


3 


Specific parent communication strategy 


1 


2 


3 


Total 







Component 3 Referral Mechanisms 



SAP is accessible to all targeted students 


1 


2 


3 




Formal referral procedures and decision-making process 


1 


2 


3 


Screening process includes clear and consistent student data collection and review 
procedures 


1 


2 


3 


Confidentiality guidelines for team are well delineated with members 
demonstrating respect for and understanding of parents’ and students’ privacy rights 


1 


2 


3 


Cases are continuously monitored 


1 


2 


3 


Total 







Component 4 Parent Participation 



Formal parent involvement procedure 


1 


2 


3 




Policy statement defining parents/guardians as decision makers who provide active 
consent and includes a formal parent involvement and satisfaction procedure 


1 


2 


3 




Confidentiality guidelines known and respected 


1 


2 


3 




Demonstration of respect for parent and family privacy rights 


1 


2 


3 




Clear and consistent parent consent process and procedures 


1 


2 


3 




Information release form process and procedures for consent to exchange 
confidential student information 


1 


2 


3 


Total 
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Component 5 Team Planning 



Regular meeting time sufficient to complete SAP work __ 


1 


2 


3 




Members’ roles and responsibilities are articulated (e.g. leaders, secretary, case 
manager) 


1 


2 


3 


Case assignment and management procedures 


1 


2 


3 


Regular meeting space with access to telephone 


1 


2 


3 


Inter-team communication system 


1 


2 


3 


Total 







Component 6 Intervention and Recommendations 



Support and provide linkages for students and parents to access school and 
community services 


1 


2 


3 




Team monitors and receives feedback on school and community assessments 


1 


2 


3 


Continuous monitoring of student progress, parent involvement and 
recommendations 


1 


2 


3 


Written guidelines for dealing with problems that are beyond the scope of the 
school’s responsibility (e.g. provision of treatment, suicidal assessment) 


1 


2 


3 


Written information available on community resources, services, and other options 


1 


2 


3 


Total 







Component 7 Follow-up and Support 



Procedures promote student access to and compliance with school and community 
services and treatment recommendations 


1 


2 


3 




School resources are available and accessible 


1 


2 


3 


Student follow-up procedure and process promotes student contact and support 


1 


2 


3 


Community resources are available and accessible 


1 


2 


3 


Continuous monitoring of student progress, parent involvement and 
treatment recommendations compliance 


1 


2 


3 


Total 







Component 8 Training 



Team members participate in all Commonwealth Approved training 


1 


2 


3 




Team members participate in a maintenance and development training program 


1 


2 


3 


Team has adequate training schedule and budget 


1 


2 


3 


Team members have opportunity and support for advanced SAP training 


1 


2 


3 


School and community agency staff participate in ongoing SAP training 


1 


2 


3 


Total 







Component 9 Outcome Indicators and Evaluation 



SAP monitoring and improvement mechanisms are in place 


1 


2 


“ 3 ' 




SAP team maintains a student data management system to track student attendance, 
GPA, failed courses, grade retention, and school leaving 


1 


2 


3 




Student SAP service participation and utilization system provides accurate and 
timely information 


1 


2 


3 




Student interventions, recommendations, and outcomes are regularly assessed for 
quality and goal attainment 


1 


2 


3 




Stakeholder input and suggestions are solicited and utilized 


i 


2 


3 




SAP satisfaction information is solicited and utilized 


i 


2 


3 


Total 
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Strategies For Using The Pennsylvania Student Assistance Program 
Component Indicator Checklist 

Any strategic application of the component indicators should take into account all of the 
individuals and/or groups who may be stakeholders in the student assistance program. The 
component areas and indicators involve many of the different stakeholders such as parents, 
students, administrators, faculty, team members, community agencies etc. One 
helpful mechanism for applying the components and indicators to a particular SAP might be the 
use of a 360 degree feedback process. This process involves getting input and feedback from all 
different levels of stakeholders, those both internal and external to your program. 

In a 360 degree process all individuals and groups who participate or benefit from your SAP 
program are asked for their perceptions of how your SAP is working. A 360 degree process 
involves not only getting feedback from those who receive service from your program such as 
administrators, faculty, parents and students but also those who provide service such as treatment 
agencies, community resources and the SAP members themselves. Because of the subjective 
nature of the feedback given, this process is not useful in determining the need for or the 
continuation of your program but, rather, in examining perceptions, trends and program 
improvement issues. 360 degree processes can open up communication with parents, students and 
within the team itself and can provide a mechanism for ongoing involvement by using the 
information gathered to formulate an action improvement plan. 

The following steps can be helpful in implementing a 360 degree feedback process for a student 
assistance program. 

• Identify the stakeholders in SAP 

• Using the components and indicators, identify the information you want to gather from each 
of your stakeholders; for example, in gathering information from parents you might want to 
focus on the indicators of access, confidentiality, perceptions of outcomes, and so forth. 

• Provide a component area and indicators checklist for each of the stakeholders to complete. 
(Different stakeholders may require different checklists to be able to gather the most pertinent 
information) 

• Process the information received. If desired, develop a rating system to summarize overall 
trends, findings and overall perceptions between and among different stakeholders 

• Develop an action improvement plan based on the feedback given, making sure that feedback 
is provided to each of the stakeholder groups/individuals. 



The following ten strategies are specific ideas for gathering information from different 
stakeholders both internal and external to your system to assist in instituting a 360 degree 
feedback and improvement process. 

Strategy One: Individual team members, working independently, define what their teams’ current 
practice is for each component and corresponding indicators. Use the Checklist to assess current 
practice levels. Gather supporting documentation for current practice. 

Strategy Two: Team members of a single building meet individually and jointly to assess the 
team’s SAP practices. Responses are compared and contrasted. Plan of action with specific 
tasks, responsibilities and timeline is developed to improve the SAP practice. 

Strategy Three: Teams within a single school district convene to share team assessment and 
action plans. Drawn from the teams’ assessments and plans is a district SAP practice 
improvement strategy and plan. 

Strategy Four : School districts within a region convene to share school assessment and action 
plans. Drawn from the schools’ assessments and plans is a regional SAP practice improvement 
strategy and plan. 

Strategy Five: County mental health and drug and alcohol program administrators and community 
agency staff (SAP liaisons as well as other county and agency staff) use the benchmarks and 
indicators to map their current SAP practice. SAP practice improvement strategies and plans can 
be developed from the point of view of the county administration, all agencies in a county, and 
individual agencies. Furthermore, agencies and programs not directly involved with SAP but 
complementary and collaborative can be included; for example. Child, Youth and Family and 
Boys and Girls Clubs. 

Strategy Six: Parents and members of the district Safe and Drug Free School advisory committee 
convene to assess SAP practice within a district. The Checklist can also be used to guide 
discussion and provide information about how SAP is practiced in the district and in a specific 
school building. 
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Strategy Seven : School staff and members of the district Safe and Drug Free School advisory 
committee convene to assess SAP practice within a district. The Checklist can be used to guide 
discussion and provide information about how SAP is practiced in the district and in a specific 
school building. 

Strategy Eight: Students, with their families and teachers, use the Checklist to assess how their 
needs are being met by SAP practice. 

Strategy Nine: State wide organizations and groups (State government, PASAP, CASSP) can use 
aggregated data from individual schools, that have differing needs, to better understand the 
challenges and opportunities facing SAP practice at the local and state level. 



Strategy Ten: Individual groups of SAP stakeholders (teachers, liaisons, principals, counselors, 
school nurses) across school buildings, districts, and the state can use aggregated data from 
individual schools, that have differing needs, to better understand the challenges and 
opportunities facing their counterparts at the local and state level. 

Thinking About The Strategies 

♦ With all strategies you can take a broad focus on all the components or a concise focus on a 

single or group of component indicators. 

♦ High performance doesn’t mean that all components and indicators are present at a particular 
level. High performance is defined by individual teams and programs and reflects a unique 
blend of component and indicator levels. 

♦ The profile of a SAP’s components and indicators is expected to change over time. They are 
dynamic. A SAP may find that certain components and indicators have priorities related to 
the developmental stage of their SAP program. For example, a new or struggling program 
might place priorities on Component 1 Policy and Procedures and Component 3 Referral 
Mechanisms. While an established and well supported SAP program may prioritize 
Component 8 Training. 
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♦ The level of SAP knowledge and experience with any given component indicator can vary 
among similar stakeholders. Likewise, expect rating variations among team members with 
different SAP roles and responsibilities. The rating variations can provide valuable insights 
into the SAP functioning and process. 

♦ In some situations, a stakeholder might elect not to rate a particular indicator (unscored 
indicators). It is recommended to note unscored indicators in reporting and tallying of the 
Checklist. 
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Developing Benchmarks For Your SAP 



The 360 degree feedback process is designed to contribute to continuous SAP improvement. 
Using the process each SAP can establish their own unique Benchmarks to guide their program’s 
development. 

Benchmarks help organizations and individuals to excel. Tucker and Codding (1998) define 
benchmarking as “The disciplined search for best practices” (p. 313). Ammons (1996) defines 
benchmarks as “the identification of best-in-class performers, the comparison of local 
performance outputs and results with those of top performers, the analysis of practices that 
account for any performance gaps, and the development and implementation of strategies to 
adjust the performance gap in one’s favor” (p. 286). The American Productivity and Quality 
Center (APQC, 2000) defines benchmark as a measured “best in class” achievement; a reference 
or measurement standard for comparison; this performance level is recognized as the standard of 
excellence for a specific organizational process. 

Once benchmarks have been established, an organization or individual may adopt procedures or 
techniques used by other organizations that they believe will be effective in their own 
circumstances. Fine and Snyder (1999) point out that the benchmarking process is ongoing. Fine 
and Snyder argue “In order to determine whether new techniques are working to achieve the 
desired benchmarks of service, organizations involved in benchmarking efforts must return to 
their performance indicators and measure the success of their work” (p. 24). If the results are not 
satisfactory, other benchmarks may be chosen. According to O’Dell (1996), benchmarking 
clearly represents the process of organizational learning. 

Using Benchmarks In Education 

The use of benchmarks in education is widespread and includes concepts of educational 
indicators and standards for students, teachers, services, programs, and schools. In all situations, 
the development and use of benchmarks focus on best practices. This search for best practices 
entails the identification of organizations, programs, and individuals that excels in its area of 
operation or practice, establishing what it is that allows it to excel, and ascertaining how to adapt 
those practices to other settings. APQR (2000) cautions organizations that “there is no single 
‘best practice’ because best is not best for everyone. Every organization is different in some way 
- different missions, cultures, environments and technologies. What is meant by “best” is those 
practices that have been shown to produce superior results; selected by a systematic process; and 
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judged as exemplary, good or successfully demonstrated. Best practices are then adjusted to fit a 
particular organization.” 

The framework for the development of benchmarks has its roots in the work of Donabedian’s 
(1980). His approach to understanding organizations and programs looks at structure, process, 
and outcomes. Using the framework benchmarks and indicators are developed and grouped 
according to their focus. For example, structural benchmarks would focus on school 
characteristics (e.g. school type, location, size), process benchmarks would focus on program 
operations (e.g. SAP) and outcome benchmarks would focus on participants (e.g. student 
behavior, academic achievement). Bottani and Tuijman (1994) discuss the construction of an 
educational indicator framework that will “provide information about the condition, the stability 
or change, the functioning, or the performance of an educational system or a part thereof. 
Education indicators may point to — or index — a certain aspect of the condition, functioning, or 
performance of an educational system, but they are not the phenomenon itself’ (p. 48). Sirotnik 
and Kimball (1999) have focused on assessment and accountability. Their work on content 
standards provides guidance for developing educational benchmarks and indicators. Using their 
work the following benchmark development guidelines are proposed. 

• The benchmarks must evaluate each school and program in terms of its own context 
as well as in comparison to other schools and programs. 

• A single indicator must not drive the benchmarks (e.g. SAP referral rate, test scores, 

, grades). 

• The benchmarks must promote equitable and substantial opportunities for learning, 
support and services for all students. 

• The benchmarks must be flexible enough to account for individual differences in 
schools. 

• The benchmarks must promote substantial, long-term professional development for 
teachers and administrators. 

• The benchmarks must not be punitive, either to students or to teachers and schools. 

• The public and the political infrastructure must support the benchmarks. 

In order to establish benchmarks that are meaningful to a specific SAP, each program must 
develop an individually tailored action plan associated with each component and indicator. The 
plan should be developed utilizing information gathered from the SAP’s reporting and monitoring 
data, the team member checklist and feedback from various stakeholders. The improvement 
process should include an assessment of the current situation, performance goals (benchmarks), 
mechanisms for measuring success and specific strategies for goal achievement. 
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How do other SAPs 
achieve this goal 
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SAP monitoring and improvement mechanisms are in 
place 


SAP team maintains a student data management 
system to track student attendance, GPA, failed 
courses, qrade retention, and leaving school 


Student SAP service participation and utilization system 
provides accurate and timely information 


Student interventions, recommendations, and outcomes 
are regularly assessed for quality and goal attainment 


Stakeholder input and suggestions are solicited and 
utilized 


(SAP satisfaction information is solicited and utilized ( 
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SAMPLE PARENT CHECKLIST 
Pennsylvania Student Assistance Program 
Component Indicator Checklist 



Name 


Date 


School 


District 



Instructions: Please rate each SAP component indicator for your SAP by using the following scale: 

1: = Never Present 2: = Sometimes Present 3: = Present All Of The Time 



Communications 



Description of SAP Services for faculty, parents, students and others including 
handbooks, brochures, etc. 


1 


2 


3 


Specific parent communication strategy 


1 


2 


3 



Referral Mechanisms 



SAP is accessible to all targeted students 


1 


2 


3 


Confidentiality guidelines for team are well delineated; with members 
demonstrating respect for and understanding of privacy rights 


1 


2 


3 



Parent Participation 



Formal parent involvement procedure 


1 


2 


3 


Policy statement defining parents/guardians as decision makers who provide 
active consent and includes a formal parent involvement and satisfaction 
procedure 


1 


2 


3 


Confidentiality guidelines are known and respected 


1 


2 


3 


Demonstration of respect for parent and family privacy rights 


1 


2 


3 


Clear and consistent parent consent process and procedure 


1 


2 


3 


Information release form process and procedure for consent to exchange 
confidential student information 


1 


2 


3 



Intervention and Recommendations 



Support and provide linkages for students and parents to access school and 
community services 


1 


2 


3 


Continuous monitoring of student progress, parent involvement, and 
recommendations 


1 


2 


3 



Follow-up and Support 



Procedures promote student access to, and compliance with, school and 
community services and treatment recommendations 


1 


2 


3 


Continuous monitoring of student progress, parent involvement and 
treatment recommendations compliance 


1 


2 


3 
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SAMPLE ADMINISTRATOR CHECKLIST 
Pennsylvania Student Assistance Program 
Component Indicator Checklist 



Name 


Date 


School 


District 



LiSti actions: Please rate each SAP component indicator for your SAP by using the following scale: 

1: = Never Present 2: = Sometimes Present 3: = Present All Of The Time 



Policy and Procedures 



Building administrators are involved and support SAP 


1 


2 


3 


Drug and Alcohol and Mental Health SAP liaisons works with team 


1 


2 


3 


SAP Coordinator for the building 


1 


2 


3 


Policy violations and consequences for alcohol, drags, involving weapons, 
tobacco 


1 


2 


3 


SAP Structure and organization (including members and titles, clear 
delineation of roles and responsibilities, meeting times, membership selection 
criteria, etc 


1 


2 


3 



Communications 



Description of SAP Services for faculty, students and others including 
handbooks, brochures, etc. 


1 


2 


3 


In-services for teachers, pupil personnel, support staff, and administrators 
provide time and support for SAP informational updates 


1 


2 


3 



Referral Mechanisms 



SAP is accessible to all targeted students 


1 


2 


3 


Formal referral procedures and decision-making process 


1 


2 


3 


Screening process includes clear and consistent student data collection and 
review procedures 


1 


2 


3 


Confidentiality guidelines for team are well delineated; with members 
demonstrating respect for and understanding of privacy rights 


1 


2 


3 


Cases are continually monitored 


1 


2 


3 



Parent Participation 



Formal parent involvement procedure 


1 


2 


3 


Confidentiality guidelines are known and respected 


1 


2 


3 


Information release form process and procedure for consent to exchange 
confidential student information 


1 


2 


3 
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Team Planning 



Regular meeting times sufficient to complete SAP work 


1 


2 


3 


Members’ roles and responsibilities are articulated (e.g. leaders, secretary, 
case manager) 


1 


2 


3 


Regular meeting space with access to telephone 


1 


2 


3 


Case assignment and management procedures 


1 


2 


3 


Inter-team communication system 


1 


2 


3 



Intervention and Recommendations 



Continuous monitoring of student progress, parent involvement, and 
recommendations 


1 


2 


3 


Written guidelines used for dealing with problems that are beyond the scope 
of the school’s responsibility (e.g. provision of treatment, suicidal 
assessment) 


1 


2 


3 


Written information available on community resources, services, and other 
options 


1 


2 


3 



Follow-up and Support 



School resources are available and accessible 


1 


2 


3 


Procedures promote student access to, and compliance with, school and 
community services and treatment recommendations 


1 


2 


3 


Continuous monitoring of student progress, parent involvement and 
treatment recommendations compliance 


1 


2 


3 



Training 



Team members have opportunity and support for advanced SAP training 


1 


2 


3 


School and community agency staff participate in ongoing SAP training 


1 


2 


3 



Outcome Indicators and Evaluation 



SAP monitoring and improvement mechanisms are in place 


1 


2 


3 


Stakeholder input and suggestions are solicited and utilized 


1 


2 


3 
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